Few ideas have enjoyed as large an impact on deep learning as convolution. For any problem involving pixels or spatial representations, common intuition holds that convolutional neural networks may be appropriate. In this paper we show a striking counterexample to this intuition via the seemingly trivial coordinate transform problem, which simply requires learning a mapping between coordinates in (x,y) Cartesian space and coordinates in one-hot pixel space. Although convolutional networks would seem appropriate for this task, we show that they fail spectacularly. We demonstrate and carefully analyze the failure first on a toy problem, at which point a simple fix becomes obvious.

Note that various models were used for experimentation and that model specifications are given in the supplementary material. It was also difficult to keep track of which kind of convolutional neural networks (CNNs) were used for which subproblem. First, this paper shows instances where CNNs fail at particular subsets of the coordinate transform problem. For "normal" CNNs, the paper shows: 1. that a fully convolutional model fails to solve the supervised coordinate classification problem and the supervised rendering problem.

Few ideas have enjoyed as large an impact on deep learning as convolution. For any problem involving pixels or spatial representations, common intuition holds that convolutional neural networks may be appropriate. In this paper we show a striking counterexample to this intuition via the seemingly trivial coordinate transform problem, which simply requires learning a mapping between coordinates in (x,y) Cartesian space and coordinates in one-hot pixel space. Although convolutional networks would seem appropriate for this task, we show that they fail spectacularly. We demonstrate and carefully analyze the failure first on a toy problem, at which point a simple fix becomes obvious.

We could use the same approach taken by many research works that generate images and paint the square with a stack of deconvolution (transposed convolution) layers. To test this idea, we created a dataset consisting of randomly placed 9 9 squares on a 64 64 canvas, as shown in Figure 1b. To evaluate how well models generalize, we define two train/test splits: a uniform split, where all possible center locations are randomly divided 80 percent/20 percent into train vs. test sets, and a quadrant split, where the canvas is divided into four quadrants: squares centered in the first three quadrants are put in the train set and squares in the last quadrant in the test set.

Which authors of this paper are endorsers? Disable MathJax (What is MathJax?)